Generally, when using a video camera or a digital still camera to photograph a color image, the incident light passes through filters for extracting certain wavelength components, such as the basic color components R (red), G (green) and B (blue). In a two-dimensional image, the imaging unit is composed of many pixels arranged in the vertical and horizontal directions. Each pixel of the two-dimensional image contains either red, green or blue color light because of the filtering of the incident light.
According to one of several alternative techniques, the type of filter is changed for every pixel and the filters are cyclically aligned in the order: R, G, B, R, G, B in the horizontal direction. This defines the color of the pixels aligned on a horizontal line of the pixel array of the sensor.
As a consequence, information of the photographed colored object is obtained only once every three pixels. In other words, an object cannot be color photographed other than in units of three pixels.
To reconstruct all the pixels of the two-dimensional image of the photographed object, it is necessary to interpolate color pixel data to obtain the color components of red, green and blue color using information contained in neighboring pixels of the pixel to be reconstructed/enhanced.
Generally, a value corresponding to the interpolated pixel is reconstructed by averaging corresponding values of a plurality of pixels surrounding the location of the pixel to be interpolated. Alternatively, the interpolated pixel may be determined by averaging the values of the pixels remaining after discarding pixels of maximum and minimum values of the neighboring pixels of the pixel to be interpolated. Also well known are techniques for detecting an edge of a photographed object by analyzing the pixels surrounding the considered cluster.
U.S. Pat. No. 5,373,322; U.S. Pat. No. 5,053,861; U.S. Pat. No. 5,040,064; U.S. Pat. No. 6,642,962; U.S. Pat. No. 6,570,616; U.S. published Patent Application No. 2003/0053687; U.S. Published Patent Application No. 2003/0007082; U.S. published Patent Application No. 2002/0101524; U.S. Pat. No. 6,366,694; European Patent Publication No. 0 497 493; European Patent Publication No. 1 176 550; and European Patent Publication No. 1 406 447 disclose techniques that are used in image processing.
Generally, the data acquired by the sensor according to a special pattern, for example the one known in the art as a Bayer color-filter array (CFA), the pattern of which is shown in FIG. 1, is characterized by associating just one of the three basic color components to each pixel. Therefore, a good quality RGB image is obtained by a specific image processing sequence (via hardware or an image generation pipeline (IGP) or via software) to generate a high quality color image. Generally, in cascade of such an image processing subsystem is associated a data compression block for reducing the band necessary for transmitting the color reconstructed image from the image processing subsystem or mass storage support, or to a remote receiver or a display unit.
Data, for example in the Bayer format, as acquired by a digital sensor, clearly represents a gross approximation of the chromatic components of the reproduced scene. It is very-importance that the accuracy of the color reconstruction via interpolation algorithms be performed on the raw data acquired by the digital sensor.
FIG. 2 illustrates a functional block diagram of a state-of-the-art image generating pipeline (IGP) for processing raw image data as acquired by a digital sensor according to a Bayer pattern to produce a high quality image in either a CIF or a QCIF format. The high quality image is to be eventually compressed by the CODEC block either in a JPEG or MPEG4 format or another similar type format for storing or transmitting the image.
The algorithms carried out by the IGP pipeline following the pre-processing block are described below, assuming for evaluation purposes, that the sensor provides a 644*484 Bayer pixel array. The IGP performs the following four sequential steps:
Defect Correction: The function of the block Def Corr is to correct various sensor damages resulting in the failure of single pixels. For the majority of applications it renders tolerable the use of sensors having a total number of single pixel defects below a certain limit. Def Corr operates a 5×5 filtering action, therefore, it loses four rows and four columns.
Color Interpolation 1: The function of the block Col Int 1 is to generate proper RGB information for each pixel of the sensor array from the Bayer pattern data. Col Int 1 operates a 5×5 filtering action, therefore, it loses four rows and four columns.
Color Interpolation 2: The function of the block Col Int2 is that of a low-pass filter. It receives the RGB pattern pixels output by the preceding block and outputs Col Int 1 RGB pixels of enhanced definition. Col Int 2 operates a 3×3 filtering action, therefore, it loses two rows and two columns.
Colored Matrix+Aperture Correction+Gamma Correction: The function of the components of this composite block may be recalled as follows.
Col Mat improves color rendition and color saturation of the image. In particular, it corrects the spectral sensitivities of the imaging sensor for enhancing chromaticity of the display in consideration of the characteristics of human sight. Col Mat has no losses of rows or columns.
Ap Corr corrects out-of-focus appearance caused by the weighed average processing, by locally enhancing contrast at contours. Ap Corr operates a 3×3 filtering action, therefore, it loses two rows and two columns.
Gamma compensates display characteristics of monitors. This is done by using a LUT (look-up-table) that can be effectively used to simultaneously correct brightness. The Gamma block has no losses of rows or columns.
Therefore, the combined processing block has a loss of two columns. Because of the various filters, a total of 12 rows and 12 columns will be lost, therefore, the final image will include 632×472 pixels. To obtain a VGA image (640×480), 8 rows and 8 columns are to be recovered.
After the application of Def Corr, 4 rows and 4 columns are recovered. After the application of Col Int 2, 2 rows and 2 columns are recovered. After the application of Apcor, 2 rows and 2 columns are recovered.
This can be arranged in different ways. In the analysis presented herein the choice has been of known IGP architectures of this type. However, they have the drawback of accounting for a burdensome overhead in reading from and writing data in the RAM at every step of the processing.
For the sample application considered:                Readings from the RAM:        
Def Corr: [(5×644)×480]/2=772,802 words (1 word=2 short int=32 bit);
Col Int 1: [(5×640)×476]/2=761,600 words;
Col Int 2: [(3×3)×640×478]/2=1,376,640 words;
Ap Corr, Col Mat, Gamma: [(3×3)×640×478]/2=1,376,640 words;
Total readings from the RAM: 4,287,682 words                Writings in the RAM:        
Def Corr: [(640)×480]/2=153,600 words (1 word=2 short int=32 bit);
Col Int 1: [3×(640)×480]/2=460,800 words;
Col Int 2: [(3)×638×478]/2=457,446 words;
Ap Corr, Col Mat, Gamma: [3×640×480 ]/2=460,800 words;
Total writings in the RAM: 1,532,646                total number of processed pixels in each block of the pipeline        
Def Corr: (640×480)=307,200 pixels;
Col Int 1: (640×480)=307,200 pixels;
Col Int 2: 3×(638×478)=914,892 pixels;
Ap Corr, Col Mat, Gamma: 3×(640×480)=921,600 pixels;
Total number of processed pixels: 2,450,892 pixels;                Arrays shift: 0;        Extra memory requested to store the intermediate arrays: 0.        
It may be observed that there is no computational overhead. In contrast, there is a large overhead in reading and writing data at every step that causes a high number of Dcache misses. Moreover, the processed image is stored as a whole in the RAM only after the IGP ends.
FIG. 3 shows a possible hardware architecture for the IGP of FIG. 2. Data is read from the embedded RAM in raster mode with the above-specified constraints, and the hardware scheme of FIG. 3 highlights the sizes of the line memories (LM) that are required.
Some of the line memory blocks include delay circuits (LM+D) necessary to filter the data. If m*n is the size of the filter, m lines of n delays circuits are needed to make available the filter m*n data to the functional block.
The interconnections of (n−1) line memories and (m−1) delay circuits to obtain the data for a 5*5 filter are shown in FIG. 4. As may be observed from the scheme of FIG. 3, all line memories that are needed in a hardware implementation of an IGP pipeline have the length of a full line of pixels of the input Bayer image. This weighs the total memory resources that are needed.
Although, as already noted above, the Codec block that generally follows (in cascade) the IGP pipeline, commonly operates sequentially on blocks of pixels into which a full image is divided. It would appear sensible to devise an IGP pipeline that would work on blocks of pixels in which the input image could be divided in compliance with the input requisites of the Codec block, but this approach has not been so far followed. This evident dismissal of the idea of implementing an IGP pipeline that could work in a “block mode” rather than in “raster mode” is due to the stringent quality requirements that certain basic functions that the IGP pipeline is intended to perform. In particular, the color information reconstruction by interpolation algorithms of pixels is a basic function to perform.
On account of an imposed uncompromised quality of color information reconstruction pixel-by-pixel from Bayer (or alike) data, if a block mode processing is considered in lieu of a raster mode processing, a substantial overlapping of the blocks of a subdivision of the pixels of the input image becomes mandatory and this will significantly increase computational overhead. This occurs because the non-linear filters (m*n) that are implemented in the IGP pipeline of the image pixels produce an output (i*j), where i<m and j<n.
Therefore, if two consecutive blocks of a subdivision of the image are not overlapped (by a quantity (m-i) in one direction or of (n-j) in the other direction), the contours of objects of the scene reproduced in the image could be lost. Consequently, the computational overhead that would be necessary to account for passing from a raster mode to a block mode architecture of an IGP pipeline practically nullifies any savings that may be achieved in terms of the total memory requirement.